Learning mixtures of structured distributions over discrete domains

Let C be a class of probability distributions over the discrete domain [n] = {1, . . . , n}. We show that if C satisfies a rather general condition – essentially, that each distribution in C can be well-approximated by a variable-width histogram with few bins – then there is a highly efficient (both in terms of running time and sample complexity) algorithm that can learn any mixture of k unknown distributions from C. We analyze several natural types of distributions over [n], including log-concave, monotone hazard rate and unimodal distributions, and show that they have the required structural property of being wellapproximated by a histogram with few bins. Applying our general algorithm, we obtain near-optimally efficient algorithms for all these mixture learning problems as described below. More precisely, • Log-concave distributions: We learn any mixture of k log-concave distributions over [n] using k ·Õ(1/ε) samples (independent of n) and running in time Õ(k log(n)/ε) bit-operations (note that reading a single sample from [n] takes Θ(log n) bit operations). For the special case k = 1 we give an efficient algorithm using Õ(1/ε) samples; this generalizes the main result of [DDS12b] from the class of Poisson Binomial distributions to the much broader class of all log-concave distributions. Our upper bounds are not far from optimal since any algorithm for this learning problem requires Ω(k/ε) samples. ∗Supported by NSF award DMS-1106999, DOD ONR grant N000141110140 and NSF award CCF-1118083. †This work was done while the author was at UC Berkeley supported by a Simons Postdoctoral Fellowship. ‡Supported by NSF grants CCF-0915929 and CCF1115703. §Supported by NSF grant CCF-1149257. • Monotone hazard rate (MHR) distributions: We learn any mixture of k MHR distributions over [n] using O(k log(n/ε)/ε) samples and running in time Õ(k log(n)/ε4) bitoperations. Any algorithm for this learning problem must use Ω(k log(n)/ε) samples. • Unimodal distributions: We give an algorithm that learns any mixture of k unimodal distributions over [n] using O(k log(n)/ε) samples and running in time Õ(k log(n)/ε4) bitoperations. Any algorithm for this problem must use Ω(k log(n)/ε) samples.